Not Too Hot, Not Too Cold: The Bundled-SVM is Just Right!

The Support Vector Machine (SVM) typically outperforms other algorithms on text classification problems, but requires training time roughly quadratic in the number of training documents. In contrast, linear time algorithms like Naive Bayes have lower performance, but can easily handle huge training sets. In this paper, we describe a technique that creates a continuum of classifiers between the SVM and a Naive Bayes like algorithm. Included in that continuum is a classifier that approximates SVM performance with linear training time. Another classifier on this continuum can outperform the SVM, yielding a breakeven point that beats other published results on Reuters-21578. We give empirical and theoretical evidence that our hybrid approach successfully navigates the tradeoffs between speed and performance.